Title: user behavior analysis - a food startup

Author: Xia Cui

Introduction

This report presents the analysis of user behaviour for a food products company. The data was collected from the company app, containing information on user actions over the span of two weeks.

One main goal of the analysis is find out where there might be potential barriers to users making a purchase,by studying the company's sales funnel.

We will also examine the results of an A/A/B test, for which the two control groups get the old font of the app, and the experiment group get the new font.

In what follows, we will first import, check and prepare the data for analysis. Then we will take a look at the number of events and users in the logs, as well as the period the data covers. Following this, we will further examine the frequency of events occurrence, user proportion of the events, the sequence of user actions, and user behaviour in the sales funnel.

Finally, we will study the results of the A/A/B test to see if the change of the fonts of the entire app will change the user behaviours. we will first look at the two control groups and see if the splitting of the groups is done properly. Then we will compare the results of the experiment group with those of the control groups.

Table of content

Part 1: Data importing

Part 2: Data preprocessing

Part 3: Event and users

Part 4: Study of sales funnel

Part 5: Analysis of experiment results

Part 6: Findings and conclusions

Part 1: Data importing

Let's get the data! To optimize data importing, we will check the first 50 rows first.

Yes indeed these two are categorical columns. We will change the datatype of 'EventName' and 'ExpId' to 'category' upon importing the data.

This is only the max of the first 50 rows. However even the unix time of year 2030 doesn't exceed the max int32 value (2147483647). We will change the data to 'int32' as its data type.

Columns look mostly fine, except the two columns that should be of 'category' data type, and the int type for the timestamp column. The columns names are also unconventional. Let's change that when importing the whole dataset.

Let's now first check the general information of the data.

Let's now check for null values and duplicates.

There are no null values, but 413 duplicates. Let's take a look.

Yes indeed. These are identical. We will drop these in the next section.

These duplicates happen at payment successful and cart, which is actually possible. We will leave this type of duplicates untouched but keep this in mind when working with the sales funnels.

Part 1 Summary

Data checking reveals a few issues we will address next in preprocessing:

Part 2: Data preprocessing

In this section, we will address the data issues as identified in the previoius section.

Addressing duplicates

No duplicates now. Awesome!

Changing event_name values

Let's have a look at these values again.

We will name the values using the keyword in its lower case.

Now let's check the names again.

Changing the event_time

Let's change the event_time to datetime dataype.

Columns look good! While we are here, let's also add a event_date column.

Interesting, now the event_name has gone back to 'object'. It might have to do with when we change the values of the column. Let's change it back.

Part 2 summary

In this section, we addressed the data issues identified earlier. Now the dataset is ready to be further examined.

Part 3: Studying the data

In this section, we will analyze the events data to address the following key questions:

First let's take another look at what our data looks like now.

First, let's take a look at how many events and users are in the logs.

3.1 Events and users

There are 5 types of events, totalling 243713 occurrences. Now let's have a look at how many users in total, and how many users in each of the 3 groups. We will count the unique users in this case.

On average, each user performed 32 event actions. Although this gives only a very general picture, considering there are only a total of 5 events, and the data covers a period of two weeks, the app is doing not bad!

Next, let's also take a quick look at the number of users from each of the three groups performing the 5 event actions.

From the above, we can see that, overall, the main page has the most share of users, followed by offers, cart, payment, and then tutorial last.

This is interesting. while from main, to offers, to cart, to payment is likely to be the sequence of actions many users follow, tutorial is unlikely to be the last user action. The extremely low user number in tutorial across the three groups only indicate the lack of popularity of the page.

We will further examine the sequence of events in the funnel section later in the report.

3.2 Time period for the logs

Let's find out the earliest and latest time of the records in the logs.

The logs covers the time period of approximately two weeks. Now let's have a look at the distribution of datetime in the logs.

The distribution of datatime is not even. If we hover the mouse to check the values, we can see that the distribution changes from very sparse to stably around 15k somewhere around July 31 and August 1. It is possible that older events could end up in some users' logs for technical reasons, and this could skew the overall picture. We need to find the moment at which the data starts to be complete and ignore the earlier section.

Just to be sure where this should be, let's also plot the distribution of the date.

Okay. Now we can be fairly certain that it is from August the 1st that the data became complete. Let's disregard the data earlier than this.

Okay. Now we have the logs spanned over approximately a week. Let's see how much data we have lost.

We lost slightly over 1% of data. It's not bad. Let's also check event wise, how much data we have lost.

For each of the 5 events, we lost approximately 0.5% to 1.5% of the data. This is relatively even and acceptable. Finally, let's have a look at how many users and events we have lost for each experiment group.

For the number of events, the loss for the two control groups and experiment group is very similar and small, so is the number of unique users for each group. The new dataset is in just as good shape!

Part 3 summary

In this section, we studied the events and users. One thing we've noticed is the sneak in of the older data into the datasets, which we then removed. In the next section, we will examine the event funnel!

Part 4 Event funnel

In this section, we will address the follow key questions:

4.1 Events and their frequency

We can see from the above that accessing the main page is the most popular event, occuring approaching 120k times. Next is offers, dropping to 46.3k occurrences, then cart (42.3k), and payment (33.9k). The least popular event is tutorials, happening only 1005 times.

4.2 Events and their users

From the bar chart above, we get the proportion of the unique users for each event. Those users are those who performed the event at least once. We should note that the proportions doesn't add up to one because the users doesn't necessarily complete just one event.

Similar to the event occurrence we calculated earlier, main page is the most popular among the users, with over 98% of the users. This is followed by offers, acccessed by over 60% of the users. Almost 50% of the users put something in the cart, and almost 47% of the users completed the payment process.

The above gives us a general picture of the popularity of each event among the users. The biggest drop user popularity happens from main to offers. However, the order manifest in popularity is not necessarily the sequence of events. Not all users follow the same sequence of events after all. To find out where the most users were lost, we will need to look at the sales funnels later in this report.

4.3 Sequence of action

First, let's take another look at the dataset.

To get the sales funnel, we will find the time the users first performed each action first.

Even a quick look at the first 10 rows show a big variety in user behaviours. Most users start with the main page, except one who start with the tutorials, then some access the offers first before the cart, whereas others access the cart first then offer, or even the payment before offers, which is a bit late to check the offers only then!

Out of the five who made payments, 4 of the users have the same cart and payment_successful time. Without further information about the app, we don't know why this happens. Perhaps the cart and payment completion page are actually one page? In real life, this is worth finding out from the app.

Out of curiosity, before we decide on the sale funnel, let's find out the percentage of users who start with each of the five events. For this mission, we are only interested in where the users started, regardless of the sequence after that.

Approximately 40% of the users start their use of the app on the main page.

29% of the users landed on the app first on the offers page.

Close to 20% of the users start with cart in the app, possibly at the same time as the payment (given the '>=' sign).

Interesting, slightly over 22% of the users start with the payment on the app, possiblity at the same time as cart (given the '>=' sign).

0.5% of the total users who started at the tutorial of the app.

The information above gives us some idea where the users first landed on the app. Potentially we can track each possible sequence to see which one is the most profitable. However, given the space of this report, we will select to study three funnels. As the main page is where the highest percentage of users first landed, we will first study two funnels:

Out of curiosity, let's also check what happens when users start with tutorials. So we will add another one to examine.

One thing to note also is that the time for payment_successful could be the same as cart, which is possible. So we will take that into account in the calculations below:

4.4 Event funnels

Funnel one: main -> offers -> cart -> payment_successful

AS shown above, for users in this sales funnel, from offers to cart is where we lost the highest percentage of user, approximately 57.9% were lost at this stage.

A total of 18% of the users made it from the beginning to the payment completion.

Funnel two: main -> cart -> offers -> payment_successful

For users in this sales funnel, from offers to payment is also where we lost the highest percentage of users (82.8%). Maybe the offers are not atttractive enough! This is followed by the stage from cart to offers (56.9%).

Only a total of 3.4% of the users from the main page made it to the payment.

Funnel three: tutorial -> main -> offers -> cart -> payment_successful

For users who watched tutorials, then accessed main, offers, cart, and payment, where users were lost the most is from offers to the cart (43.9%). As high as 32% of users who watched tutorial ended up paying successfully!

Part 4 summary

In this section, we studied the events and users. In terms of both event occurrence and proportion of users, the main page has the highest popularity, followed by offers, cart, payment, and tutorials.

We found that that users vary a lot in where they started their journey on the app, and selected 3 sales funnels to calculate the proportion of users who progress from one stage to the next. For all three, we lost the highest percentage of users after the offer page. It's hard to say why but it would help to check the offers page to make sure these are attractive and encouraging, as well as link to the cart or payment page smoothly.

Having said that, we only looked that the users who started with main or tutorials. There are users who started somewhere else, or started with main but made shortcut and nevertheless made payments successfully. It's important to keep this in mind.

Among the 3 funnels, in the one users started with tutorials (funnel 3) we have the highest percentage of users who eventually made the payment (32%). Those who followed the sequence of main, offer, cart, and payment came second in terms of the proportion of users among the inital ones who made payment (18%).

This could indicate that those who would start with tutorials are more keen to purchase in the first place, or the watching tutorials in a way might have facilitated the eventual payment. It's hard to tell. We also need to be aware that those who start with tutorials only represent a very small population among the total users (0.05%).

Part 5: Analysis of experiment results

In this seciton, we will address the following key questions:

5.1 A/A comparison

Ideally, the number of users in the two the A/A group doesn't vary by more than 1%. In this case, we have the difference just slightly over 1%. It can be considered acceptable.

Next, let's create the subsets from the data containing each of the control groups, the experiement group, and the control groups combined.

Let's also check if the users accidentally show up in more than one group.

No overlapping, great!

For the hypothesis testing that the proportions of users who performed the event among the total users are the same between two groups, we will define a function, which passes the name of the event, and which two groups as parameters.

For the test performed by this function, the significance level is set as 0.05.

The null hypothesis is that the two proportions of users performing the actions among the total users are the same; the alternative hypothesis is they are different.

We will first run the test without correcting the significance level.

Now we will run the function looping through the list for the two control groups.

As shown above, we fail to reject all the null hypotheses above. Given that we are performing multiple testings using partially same data, in theory we should adjust the significance level.

However, in our case, we fail to reject all the null hypothesis already. Adjusting the significance level wouldn't change the result.

The results above confirm that the splitting of the groups is done properly. We can move on to the A/B test.

5.2 A/B comparison

For A/B test, we will use the same function defined earlier. The significance level is still 0.05, and we will perform the test between B group and each of the control groups, and then between B group and the control groups combined.

Let's first perform the test for events between the first control group and B group.

We fail to reject any of the null hypotheses above. Now let's run the test for events between the 2nd control group and B group.

The same happened. We failed to reject any of the null hypotheses.

Finally, let's run the test between the control groups combine and B group.

Comparing the B group against the control groups combined didn't make any difference. We failed to reject all the null hypotheses. Changing the font of the app didn't make any statistically significant different to the share of users who perform any of the event actions.

Technically, we performed multiple testings above: A1 and B, A2 and B, and A and B, and within each group, we performed 5 tests using partially same data. With multiple testing, the probability of making a type I error increases with each new hypothesis test, which is the probability of rejecting the null hypothesis when null hypothesis is true.

In our case, none of the test results in rejecting the null hypothesis. Therefore we don't need to apply any of the methods to correct the significance level.

The siginificance tests in this section show that we can not prove that changing the font of the app will change the user behaviours. We don't have enough evidence to say that the users are intimidated by the new fonts, where were the concerns of the managers.

Part 6: Findings and conclusion

In this report, we studied the user log data of a food company app over the span of one week.

The investigation of event occurrence shows that the main page is the most popular among the users, whereas tutorial is the least. Users show great variety in terms of where they first landed on the app, and follow different sequence of actions.

To examine the user behaviours, we examined three sales funnels, regardless of the different sequence of actions, the funnels are shown to have lost the highest percentage of users after the offers page. The company will need to investigate the offers page to see if, for example, the offers are attractive enough, or the offers link back to the cart or payment page smoothly.

We also examine the results of the A/A/B test to see if the change of app fonts have changed the user behaviour. Outcomes of the tests show that there is no statistically significant differences in user behaviour towards the old and new version. In other words, we can not say that users are intimidated by the new fonts.